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ABSTRACT 

A qualitative study investigated the test-taking 
behaviors, knowledge, and perceptions of 20 urban, adult basic 
education students reading at third to fifth grade equivalency 
levels. The entire reading comprehension subtest of the Test of Adult 
Basic Education, levels E and M, was administered under standardized 
conditions, A combination of observation and self-report 
methodologies was used. The researcher observed the participants 
during the test and asked some questions raised by observations after 
the test, A few days later, the researcher asked the participants to 
give advice to a hypothetical test-taker and to read each passage 
aloud and choose answers orally without referring to previous 
answers, A comparative analysis was made of the data. Findings 
indicated that levels E and M lacked construct validity because the 
majority of test-takers were penalized for interacting with the text, 
interpreting it, and bringing their own knowledge and experience to 
bear upon the question-answering tasks. Two other factors also 
heavily influenced the validity of participants 1 

scores — test-wiseness and speed. Problems of test design adversely 
affected participants' ability to demonstrate their reading 
comprehension skills. The test-takers demonstrated a high degree of 
metacognitive awareness and self-regulation, (YLB) 
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THE TEST-TAKING BEHAVIORS, KNOWLEDGE, AND PERCEPTIONS 
OF ADULT BASIC EDUCATION STUDENTS ON A STANDARDIZED 
READING COMPREHENSION TEST 

Multiple- choice standardized tests of reading comprehension are 
increasingly being used in making decisions about whether to hire people for 
jobs, offer them job-training, or enroll them in educational programs. it is 
ironic that this is happening at a time when many professionals in the field 
of reading are questioning whether such tests are valid measures of reading 
comprehension (Farr & Carey, 1986; Valencia & Pearson, 1987; Wixson & Peters, 
1987) . Concerns about whether these tests measure what they purport to 
measure can only be answered when there is more data about what adults think 
and do as they take these kinds of tests. 

Purpose of the Study 
The purpose of this qualitative study was to take a comprehensive as 
well as detailed look at the test- taking behaviors, knowledge, and perceptions 
of a little studied population- -urban, adult basic education students reading 
at third to fifth grade equivalency levels. The test used was the reading 
comprehension subtest of the Test of Adult Basic Educat ion (TABE) , the most 
widely used adult basic skills test in the United States (Jackson, 1990; 
Sticht, 1991) . The study sought answers to three questions: 

1. How did participants perceive the purpose of the test and its task 
demands? 

2. What were participants' observable test-taking behaviors? 

3. How did participants explain their test-taking behaviors? 

The view of the reading process upon which this study was based is 
defined as the process of negotiating meaning between a reader and an author 
through the medium of a text. This understanding of the nature of the reading 
process has necessitated a new look at the ways reading comprehension have 
traditionally been evaluated. 
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Method 

In contrast to previous studies which have used selections from tests 
under untimed conditions, this ethnographic study made the testing situation 
as naturalistic as possible by administering the entire reading comprehension 
subtest of the TABE, levels E and M, under standardized conditions including 
time limits and published answer sheets. 

The 20 participants were students in a community-based adult literacy 
program in a large midwestern city. The twelve women and eight men ranged in 
age from 23 to 64. Sixteen were African-American; one was Hispanic; and three 
were European-American. Eight had completed a Laubach Literacy program before 
enrolling in the adult basic education class. Half of the participants had 
grown up in rural areas in Southern states and in Appalachian areas of West 
Virginia and Pennsylvania. 

I used a combination of observation and self -report methodologies. In 
the first interview I shadowed the participant as she or he took the test and 
observed such things as eye and hand movements, time spent reading each 
passage and question, accuracy with which answers were recorded on che answer 
sheet, and glances at the clock. After the test, I asked a few questions 
raised by my observations. A few days later, in a second interview, I asked 
the participant to give advice to a hypothetical new test-taker about the best 
way to take this test. Th n I asked him oi her to read each passage aloud and 
choose answers orally without referring to previous answers. In other words, 
participants took the test twice, once under standardized conditions and once 
orally. After each answer the student was asked, "How did you decide that 
this was the best answer?" 

Data collection and data analysis occurred simultaneously. A 
comparative analysis was made of three slices of data: (1) observation notes 
taken during the administration of the test, (2) the test documents, and 
(3) transcriptions of the interviews. Using the constant comparative method, 
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I compared successive cases, identified patterns, and revised them as the data 
indicated. Four peer reviewers verified the reliability of the patterns I had 
identified. 

Findings 

Perceptions of Test Purpose 

Based on frequency, there were three main categories of response: 
(1) A typical response from eight participants was "How well I can read it and 
understand what I read." Five of these students added: "Get the right 
answer." It is interesting to note that all of these students based most of 
their answers solidly on information in test passages. (2) A typical response 
from seven other participants was "To see what your level is." Four of these 
students over- relied on their prior knowledge when explaining a number of 
their answers. (3) A typical response from five participants was "Knowledge 
of what I know and what I don't know- -my learning and education." Four of 
these students over- relied on their previous knowledge and ignored information 
in test passages when explaining from 6 to 19 of their answer choices. They 
seemed to view the test more as a measure of their knowledge than of their 
ability to comprehend what they read. 

It is interesting that participants who had a clear understanding of the 
purpose of the test based their answers on information in test passages 
whereas those whose understanding of the test's purpose was less clear tended 
to base a larger number of answers on their own knowledge and opinions. This 
finding supports a statement by Peter Johnston (1983) that "Part of the 
reader's task is to figure out exactly what the goal of the examiner is- -what 
he or she wants to know or to hear." Clearly some participants did not 
understand what the examiner wanted to know. 
Sample Passage 

Three test -takers showed signs of confusion and disorientation as soon 
as they turned the page containing the sample passage and discovered that the 
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first page of the test was formatted differently than the sample page. 
Reading the Passages 

All but one of the participants started on page one and worked 
consecutively to the last page. They explained that they did not realize they 
were allowed to skip over difficult passages and questions. Some felt they 
might have had difficulty recording their answers accurately if they had 
skipped around. 
Reading the Questions 

Participants taking the easier level of the test often had problems with 
some of the words and phrases the test routinely uses to frame questions 
(e.g., according to the passage , phrase ) . Some did not know the meaning of 
the word passage ("It's in the Bible," "Someone who rides with you") . 
Answering the Questions 

Participants engaged in sophisticated reasoning processes which they 
were usually able to articulate. However, they were unable to offer equally 
cogent reasons for rejecting alternative answers. This was not surprising 
considering that the majority of participants had indicated that they had 
little or no previous experience in taking multiple-choice tests. 

Many participants interpreted certain passages and questions differently 
than the interpretation required by the test, and they were able to justify 
their, interpretations. For oxample, a passage about the French impressionist 
painter Monet stated that he painted many pictures of the water lilies in the 
pool in his garden. Test -takers were asked to complete the following 
statement: "Monet often painted outdoor scenes because he probably..." (Keyed 
answer: "loved nature") (p. 12, Level E, Form 6). Two students preferred the 
distractor "had a big garden" because if he had a garden with a pool in it, it 
must have been a big garden. When one thinks in terms of backyards in the 
central city, this is a logical assumption. But why did students reject the 
keyed answer? To the probe: "When you think of nature, what do you think of?" 
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they answered as follows: Brenda : "The woods, the trees... I don't think of 
gardens.: Susan: "Ain't no nature! It's about his garden." Both students 
appeared to define nature as wild and natural, untouched by humans. 

Test- takers' alternate interpretations were not always idiosyncratic. 
Several passages elicited variant interpretations from the majority of 
participants . One of the maj or causes of incorrect answers was over- reliance 
on personal knowledge in preference to information in the text . Participants 
gave more reader-based answers in response to test passages on familiar topics 
(e.g., shopping on a limited budget, a letter of complaint about a defective 
car) than they did on more academic, unfamiliar topics (e.g., Monet's 
paintings, the history of the umbrella) . This confirms the findings of a 
study by Hill and Perry (1988) which compared adult basic education students' 
answers on the 1976 version of the TABE with their answers on the revised 1987 
version. In the 1987 version the test-makers included some passages designed 
to be more relevant to adult experiences. However, Hill and Perry reported 
that the new version confused test -takers more than the old version. The new 
version's more interesting and relevant passages stimulated readers to relate 
text to the world outside the passages and to draw upon their own knowledge 
and experience. However, its questions still required readers to recycle 
information in the passages. This conflict between the reading task and the 
question task led students who lacked test- taking skills astray. They were 
especially apt to give reader-based answers when the information in a passage 
contradicted their own experience. 
Recording Answers on the Answer Sheet 

Three participants had major problems with tracking visually from the 
test booklet to the bubble answer sheet . One such student recorded only 9 
answers correctly, but orally answered 21 questions correctly. Nearly half 
had some minor difficulty recording their answers. These errors resulted in 
the loss of a point or in lost time spent erasing and re- recording some of 
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their answers. The design of the answer sheets contributed to students' 
difficulties. For example, the columns and question numbers on the answer 
sheet did not correspond to the columns and question numbers on the test 
booklet . 

There were surprisingly few answer changes. Nine participants made no 
changes and another five made one or two changes. Two test-takers said that 
they thought they were not allowed to erase. Brenda : "In most tests ... they 
don't want you to erase and smear up the paper and get it all dirty lookin' . 
When I was in school, they didn't allow you to erase. If you picked a answer, 
and you wanted to change it, it was just too bad. You just got that wrong." 
Others said they did not change answers because of time limits. 
Finishing Withi n Time Limits 

Only 45% were able to complete the 40 questions in the allotted 37 
minutes, and two of those who completed the test did so by ceasing to read the 
passages when they saw that time was running out. It is important to note 
that the Educational Testing Service considers a test to be speeded if fewer 
that 80% finish the test (Davis, Kaiser, & Boone, 1987)., and Morante (1989) 
has argued that 90% should be able to finish a test. 

The speeded nature of the TABE particularly ponalized slow, deliberate, 
thoughtful people. May complained: "I have to read and then think, you know. 
I can't just shout out the answer, just off. I have to think about it, and 
that's sort of put a bind on me ... " 

All but one of the participants felt strongly that random guessing in 
order to finish on time was wrong. Mike: "I would not guess! Guessing is 
not helping you any." Susan: "I wouldn't guess... It isn't right because you 
should properly read it and then figure out what the answer is . . . It 1 s like 
cheating! You're not doing what you're supposed to do." 

When allowed to finish the test untimed, participants did not change 
their test-taking behaviors significantly and took only 2 to 15 extra minutes. 
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Conclusions 

The first major conclusion of this study is that levels E and M did not 
give a valid picture of the reading comprehension proficiency of the majority 
of the participants. If reading is viewed as a process in which the reader 
actively searches for meaning by integrating information from text with his or 
her own experience and knowledge, then this test lacks construct validity 
because the majority of test-takers in this study were penalized to a greater 
or lesser degree for interacting with the text, interpreting it, and bringing 
their own knowledge and experience to bear upon the question-answering tasks. 

Two other factors also heavily influenced the validity of participants 1 
scores--test-wiseness and. speed. As a result, the test scores of the majority 
of participants gave a lower estimate of their reading comprehension ability 
than their oral performance on the test indicated. Their reluctance to use 
random guessing to finissh within time limits further lowered their scores. 

Three problems of test design also adversely affected participants' 
ability to demonstrate their reading comprehension skills: a sample passage 
whose format differed from the format of the actual test, answer sheets whose 
columns and numbers did not correspond to the columns and numbers of the test 
booklet, and unfamiliar words used to frame questions. 

A second major conclusion is that the ABE test -takers who participated 
in this study demonstrated a high degree of metacognitive awareness and self- 
regulation. Both high scorers and low scorers could explain what they did and 
why they did it. In addition, their comments and explanations revealed them 
to be motivated readers with a strong sense of integrity as learners. 

As a teacher researcher, the most important thing that I learned from 
doing this study was the value of listening to students as they explained how 
they chose their answers. I am convinced that any valid assessment instrument 
which attempts to measure reading comprehension in order to inform instruction 
must include evaluation of the process by which students get their answers as 
well as evaluation of the answers themselves. 
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